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AIRBORNE  WARNING  AND  CONTROL  SYSTEM  (AWACS) 
INTELLIGENT  TUTORING  SYSTEM  (ITS) 


BACKGROUND 

The  United  States  Air  Force  (USAF)  at  Tinker  AFB,  Oklahoma,  trains  USAF  officers 
as  Weapons  Directors  (WDs)  to  serve  aboard  the  E-3B/C  Airborne  Warning  and  Control 
System  (AWACS)  aircraft.  The  AWACS  platform  supports  Command,  Control  and 
Communications  (C®)  missions  in  the  airborne  environment.  Armstrong  Laboratory,  Crew 
Technology  Division,  Sustained  Operations  Branch  (AL/CFTO),  in  cooperation  with 
Systems  Research  Laboratories,  Inc.  (SRL),  configured  WD  crewstations  and  affiliated 
systems  in  the  Aircrew  Evaluation  Sustained  Operations  Performance  (AESOP)  facility  to 
run  defensive  counter  air  (DCA)  mission  simulations.  Teams  of  three  WDs  detect,  identify, 
intercept,  and  destroy  enemy  aircraft  attempting  to  attack  friendly  forces  or  penetrate 
friendly  airspace.  A  Senior  Director  (SD)  serves  as  the  immediate  supervisor  in  the  chain 
of  command.  In  a  DCA  mission,  the  SD  oversees  and  coordinates  WD  efforts  to  execute 
the  directives  of  the  senior  battle  staff. 

In  cooperation  with  the  Technical  Training  Research  Division,  Human  Resources 
Directorate,  Armstrong  Laboratory  (AL/HRT),  SRL  began  research  and  development  of 
an  Intelligent  Tutoring  System  (ITS)  in  the  Air  Force  Specialty  Code  (AFSC)  17XX  C® 
domain.  The  effort  was  to  provide  a  proof  of  concept,  with  recommendations  for  future 
development. 


DEVELOPING  CRITERIA 

One  of  the  first  tasks  was  to  narrow  the  instructional  domain.  The  following  criteria 
were  established: 

1.  The  domain  was  restricted  to  that  of  weapons  controllers:  WDs,  SDs, 
Weapons  Controllers  (WCs),  and  Weapons  Assignment  Officers  (WAOs). 

2.  Due  to  practical  limitations  on  equipment  and  funds,  interactive  voice 
communications  were  not  included. 

3.  The  domain  included  areas  that  would  fill  an  immediate  need  of  the  Air 
Force’s  17XX  operational  training  community. 
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4. 


The  domain  was  limited  in  order  to  accomplish  development  in  the  time 
available. 

5.  The  ITS  was  restricted  to  Initial  Qualification  Training  (IQT).  By  focusing  on 
initial  training,  changes  in  performance  would  be  more  dramatic  and, 
therefore,  easier  to  measure. 

« 

6.  Finally,  in  order  for  the  ITS  to  teach  something  more  than  a  routine 
procedure,  we  wanted  to  incorporate  some  decision-making  skills. 

Using  these  criteria,  training  systems  at  Tyndall  AFB  and  Tinker  AFB  were  deemed 
appropriate  for  adapting  to  an  ITS.  At  Tyndall  AFB,  all  17XXs  receive  their  initial  Air  Force 
training  in  the  weapons  controller  career  field.  After  graduating  from  Tyndall’s 
introductory  course,  some  17XXs  continue  to  Tinker  for  initial  WD  training  in  AWACS. 

After  initial  contacts  with  training  sites,  we  chose  to  focus  on  Tinker’s  AWACS 
community.  After  a  fact-finding  trip  to  Tinker  AFB  (11-13  August  1991),  the  main  subject 
area  for  initial  development  was  identified  as  Block  I,  AWACS  WD  Initial  Qualification 
Training.  Appendix  A  lists  the  topics  and  switch  actions  covered  in  Block  I  during  the  first 
13  days  of  WD  training  at  Tinker  AFB.  This  list  is  taken  from  the  Initial  Qualification 
Training  Reading  Guide  (6  June  1990).  This  list  is  an  introduction  to  the  switch  actions 
a  WD  uses  in  performance  of  job  tasks.  Block  I  IQT  met  the  first  five  criteria  for 
developing  an  ITS.  The  Director  of  Operations  of  the  552d  Tactical  Training  Squadron 
also  indicated  that  an  ITS  for  this  Instructional  block  would  widen  a  traditional  training 
bottleneck  for  AWACS  WD  training,  thus  fulfilling  criterion  #  3. 

Criterion  #  6,  decision-making  skills,  was  not  met.  However  this  criterion  was  of 
a  lower  priority  for  two  reasons.  First,  only  Block  I  IQT  WD  instructional  materials  were 
set  and  the  AWACS  WD  training  program  was  being  rewritten  to  take  into  account 
lessons  learned  from  Desert  Shield/Storm.  Second,  there  is  an  ongoing  reorganization 
by  the  Air  Force  in  the  training  of  17XXs. 


PREPARATION 

While  at  Tinker,  SRL  tried  to  ascertain  how  Instructor  WDs  (IWDs)  taught  and 
evaluated  IQT  WD  students-particularly  as  related  to  Block  I  IQT  WD  training.  Important 
findings  included: 

1.  Time  for  task  completion  is  ill-defined  In  early  instructional  blocks. 

2.  Number  and  types  of  errors  are  also  ill-defined. 
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3.  Window  for  task  completion,  while  not  critical  at  first,  rapidly  becomes  more 
important  than  time  for  task  completion.  The  size  of  the  window  remains 
ill-defined. 

4.  Fighter  Weapons  School  is  considered  more  important  for  good  experience 
in  initial  categorization  of  student  abilities  and  capabilities  than  any  other 
prior  experience. 

5.  Students  are  not  accelerated  through  training,  even  if  they  show  exceptional 
skill  or  aptitude. 

6.  Prior  experience  and  training  records  play  a  part  in  evaluating  students,  but 
as  an  exception,  not  a  rule. 

7.  Only  some  IWDs  used  a  dual-tracking  task  in  training  of  Block  I IQT  WDs, 
but  this  technique  was  being  institutionalized  as  part  of  the  formal  program. 
The  dual-tracking  task  consists  of  a  single  piece  of  symbology  on  a  single 
track  in  an  oval  orbit.  The  student  is  required  to  keep  the  symbology  on 
the  track  throughout  the  lesson  and  while  performing  the  procedures.  This 
dual-tracking  task  is  not  objectively  measured,  but  subjectively  indicates  to 
the  IWD  that  the  student  WD  learned  the  lesson  well. 

8.  Most  switch  actions  are  taught  and  used  for  two  simulations  in  a  row,  and 
then  not  trained  or  used  until  a  student  progression  test  is  administered. 

9.  Most  student  evaluation  is  subjective. 


BENEFITS  OF  AN  ITS 

ITS  can  eliminate  or  ameliorate  many  obstacles  to  efficient  and  effective  training, 
including  those  previously  mentioned.  ITS  can  take  full  advantage  of  quantitative 
measures  developed  in  cooperation  with  expert  IWDs,  allow  individually  paced  escalation 
of  complexity  and  volume  of  tasks,  and  implement  consistent  application  of  evaluation 
criteria,  in  addition: 

•  ITS  allows  better  use  of  manpower  by  enabling  an  IWD  to  manage  the 
training  of  many  student  WDs  rather  than  two  or  three.  At  the  same  time, 
ITS  effectively  allows  a  one-to-one  student-to-teacher  ratio. 

•  Because  of  the  self-paced  aspect  of  ITS,  training  can  proceed  at  a  faster 
rate  for  more  able  or  experienced  students  and  allow  less  able  students 
ample  practice  time  and  opportunities  to  hone  their  skills. 
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As  the  need  for  rapid  response  escalates,  a  WD  must  increase  the  work 
pace.  This  increase  can  occur  only  where  there  is  spare  capacity.  The 
optimum  area  for  this  increase  is  that  of  console  operations.  ITS  can 
objectively  judge  this  spare  capacity  and  thereby  enhance  WD  training  by 
developing  high-performance  knowledge  and  skills  in  console  operations. 
ITS  simulations  can  foster  achievement  of  these  skills  through  "consistent 
practice"  (Regian,  1990). 

ITS  allows  a  student  WD  to  consistently  build  on  recently  acquired 
knowledge  and  skills  rather  than  moving  on  to  other  tasks  before  these  new 
skills  are  reinforced. 


DESIGNING  THE  ITS 


ITS  Paradigm 

The  second  task  under  this  effort  required  the  creation  of  a  conceptual  paradigm 
of  an  ITS.  The  five  important  elements  that  must  be  related  to  each  other  are: 

•  a  Simulation  System, 

•  a  Knowledge  Domain, 

•  an  Instructor  Model, 

•  a  Student  Model,  and 

•  an  Intelligent  Interface  that  links  them  together  (Burns  &  Parlett,  1990). 
These  elements  are  graphically  represented  in  Figure  1 . 
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Instructor  /  /|  Model 


Figure  1.  Intelligent  Tutoring  System  Paradigm 


Each  of  these  five  elements  is  interrelated  to  the  others.  A  development  in  one 
element  often  requires  a  corresponding  development  in  one  or  more  of  the  others  to 
remain  linked  as  a  coherent  whole.  For  example,  the  procedural  steps  for  executing  the 
Commit  switch  action  must  be  included  in  the  relevant  Knowledge  Domain.  The  capability 
for  this  switch  action  must  exist  on  the  simulator.  The  Instructor  Model  must  indicate 
how,  when,  and  where  in  the  course  of  instruction  to  teach  the  Commit  switch  action. 
The  how,  when,  and  where  of  the  instruction  depends  heavily  on  the  Student  Model.  The 
Intelligent  Interface  must  make  the  match  of  the  Student  Model  to  the  student,  pick  the 
correct  format  of  instructions  for  the  lesson,  execute  the  lesson  on  the  simulator,  and 
evaluate  the  student’s  progress. 

Knowledge  Domain 

For  practical  reasons,  the  Knowledge  Domain  element  comes  first.  Specifically, 
this  element  identified  as  the  552  AWACW  IQT  WD  Block  I,  Weapons  Director  Student 
Study  Guide,  Volume  1,  of  training  coursj  E3000BQODX.  This  guide  introduces  students 
to  AWACS,  the  WD  Multi-Purpose  Console  (MPC),  and  several  switch  actions,  building 
a  declarative  knowledge  base  and  a  procedural  knowledge  base.  The  emphasis  is  on 
procedural  knowledge  base  (i.e.,  how  to  perform  certain  WD  switch  actions).  A  specific 
list  of  these  switch  actions  was  provided  by  a  representative^  of  the  552 
AWACW/DOQMW  office,  which  is  responsible  for  tfie  training  course  development  at 
Tinker  AFB  for  WDs. 


Simulation  System 

The  next  element,  the  Simulator,  is  well  developed.  C®  Generic  Workstations 
{C®GWs)  and  the  supporting  AWACS  Simulation  software  in  the  AESOP  facility  (Schiflett, 
Strome,  Eddy,  and  Dalrymple,  1990)  present  the  AWACS  simulation.  Most  of  the  switch 
actions  taught  in  Block  I  at  Tinker  AFB  are  available  on  the  C®GWs.  The  following  Block  I 
switch  actions  have  not  yet  been  implemented  on  the  C®GWs; 

Restricted  Area 
RN/DES/NTN  SD 
Add/Delete  Airbase 
Area  Define/Delete 
Corridor  IFF  SD 
Radar/IFF  Tracking 

These  switch  actions  amount  to  28%  of  the  switch  actions  taught  in  Block  I,  IQT 
WD  training.  An  estimated  280  h  of  programming  development  are  needed  to  fully 


'  Personal  communication  from  Capt.  Fowler 
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develop  these  switch  actions.  In  all  other  respects,  the  Simulator  element  closely 
emulates  the  presentation  and  functions  of  an  AWACS  WD  MPC. 


Instructor  Model 


The  instructor  Model  element  contains  instructional  goals,  accounts  for  student 
attributes,  structures  the  Knowledge  Domain,  and  presents  the  structure.  Specifically, 
these  goals  are  to  teach  the  student: 


1.  The  physical  layout  of  the  MPC  (i.e.,  where  the  switch  action  buttons  are 
locatedV 


2.  The  procedural  steps  for  executing  MPC  Console  Checkout. 

3.  The  procedural  steps  for  executing  the  following  switch  actions: 


Assign  Console 
Line 

Coordinate 
Restricted  Area 
RN/DES/NTN  SD 
Add/Delete  Airbase 
Locate  SIF 
Request  SIF 
Mode  IV 

Radar/IFF  Tracking 
Request/Assign  IFF/SIF 


TD  Index 
Circle 

Tactical  Bearing  &  Range 
Bearing  and  Range 
Initialize  Special  Point 
Area  Define/Delete 
Corridor  IFF  SD 
Initiate 
Reinitiate 
Assign/Defer 

ADS  Panel  Channel  activations 


The  goals  lay  the  foundation  for  evaluating  the  student’s  performance.  Acceptable 
execution  of  these  procedures  by  a  student  WD  indicates  successful  transfer  of 
knowledge.  However,  defining  acceptab’e  execution  includes  several  measures: 

•  time  of  complete  task  execution, 

•  time  of  execution  of  each  step  within  the  task,  and 

•  a  time  window  for  accepting  task  start  and  end,  error  toleration,  and 
alternate  step  sequencing,  when  it  exists. 


The  values  that  address  these  measures  are  not  currently  known  with  certainty. 
Therefore,  our  initial  set  of  values  for  evaluation  was  determined  by  a  Subject  Matter 
Expert  based  on  years  of  experience  as  a  WD/SD/IWD.  The  set  of  values  chosen  for 
different  students  is  based  on  student  attributes. 
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student  Model 


Accounting  for  student  attributes  requires  recognizing  appropriate  student 
differences,  setting  appropriate  goals  based  upon  these  differences,  and  then  selecting 
a  mode  of  instruction  that  best  matches  both  the  student  and  the  instructional  goals. 
Recognizing  appropriate  student  differences  relies  heavily  upon  the  Student  Model.  A  WD 
student  is  matched  to  the  appropriate  type  in  the  Student  Model.  Once  the  student  type 
is  known,  then  the  Instructor  Model  has  the  correct  set  of  instructional  goals  for  that 
student  type.  The  prototype  ITS  has  only  one  set  of  instructional  goals  based  on  the 
lowest  level  of  student  type.  Selecting  a  mode  of  instruction  impacts  how  the  Knowledge 
Domain  is  structured  and  how  information  is  presented. 


NECESSARY  WD  KNOWLEDGE/SKILLS 

There  are  three  distinct,  but  interrelated  areas  of  knowledge  and  skill  necessary  to 
be  a  functioning  and  qualified  AWACS  WD.  These  areas  are  declarative  knowledge, 
procedural  knowledge  (Barr  &  Feigenbaum,  1982;  Anderson,  1988),  and  operational  skills. 

Knowledge  about  the  system,  its  purpose,  components,  events,  and  the 
relationships  among  them  is  defined  as  Declarative  knowledge.  Examples  include  how 
radar  works,  the  different  types  of  intercept  geometry,  air  power  doctrine,  and  brevity 
code  words. 

Procedural  knowledge  deals  with  how  to  operate  the  equipment.  Examples  include 
reading  scope  presentations,  talking  and  listening  by  means  of  the  communications 
equipment,  and  executing  switch  actions. 

Operational  skills  encompass  knowledge  at  both  cognitive  and  meta  levels  (Chu 
&  Mitchell,  1991).  At  the  cognitive  level,  the  WD  must  acquire  some  high  level  skills  to 
cope  with  the  demands  of  a  complex  task  environment  to  properly  apply  the  declarative 
and  procedural  knowledge  in  both  normal  and  abnormal  problem  solving  situations 
(Woods,  1988).  Time  management  and  cost-benefit  analysis  are  two  skills  necessary  for 
the  proper  coordination  of  multiple  WD  tasks  in  a  complex  and  dynamic  environment,  with 
system  events  competing  for  the  operator’s  attention.  Other  critical  skills  resulting  from 
the  dynamic,  complex  nature  of  the  system  include  adaptiveness  and  disturbance 
management  that  are  common  to  other  complex  systems.  Two  skills  that  are  critical  to 
the  WD  world  include  dichotic  listening  and  thre^imensional  spatial  orientation  with 
motion  of  objects  while  remaining  stationary. 

Meta  skills  concern  knowledge  about  how  to  learn  effectively.  The  body  of 
knowledge  the  novice  WD  must  learn  can  be  overwhelming.  Consequently,  at  the  meta 
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level,  the  WD  needs  to  know  how  to  monitor  the  learning  process  and  manage  different 
activities  to  get  the  most  out  of  training. 

The  declarative  and  procedural  knowledge  together  form  the  knowledge  domain 
the  operator  must  have.  Operational  skill  can  be  viewed  as  the  operator’s  successful 
acquisition  and  application  of  the  knowledge  domain  during  training  that  transfers  to  the 
actual  task  environment. 


Skill  Levels 

In  each  of  the  three  areas,  a  WD  is  rated  on  a  scale  of  1-5,  with  5  being  the 
highest.  Each  numerical  rating  corresponds  to  a  label  ranging  from  1 -Naive,  2-Novice, 
3-Journeyman,  4-Expert,  and  5-Master.  The  following  is  a  description  of  each  category: 

1  NAIVE:  Indicates  a  complete  lack  of  knowledge  and/or  skill. 

2  NOVICE:  Has  some  knowledge  and/or  skill,  but  not  enough  to  operate 

independently  to  complete  assigned  tasks  in  a  timely  manner. 
Does  not  recognize  all  patterns  of  stimuli.  Does  not  know 
how  to  order  behavioral  actions  in  response  to  recognized 
patterns  of  stimuli. 

3  JOURNEYMAN:  Has  enough  knowledge  (score  85%  or  better  in  written 

exams)  and/or  skill  to  operate  independently  to  complete 
assigned  tasks,  but  not  always  in  a  timely  manner. 
Recognizes  common  stimuli  variable  patterns  and  applies 
domain  rules  of  behavioral  responses.  Does  not  recognize 
situations  in  which  the  domain  rules  do  not  apply,  and/or  in 
which  new  variables  or  patterns  exist.  Perseveres  in 
attempting  to  apply  what  is  knovAm.  Not  yet  capable  of  solving 
difficult  or  complex  problems  or  developing  new  rules. 

4  EXPERT:  Has  superior  knowledge  and/or  skill.  Operates  well 

independently  to  complete  all  assigned  tasks  in  a  timely 
manner.  Recognizes  common  stimuli  variable  patterns  and 
app'ies  domain  rules  of  behavioral  responses.  Recognizes 
existence  of  new  variables  and/or  situations  in  vMiich  the 
domain  rules  no  longer  apply,  and  can  usually  develop  a 
solution.  Capable  of  solving  most  difficult  or  complex 
problems. 

5  MASTER:  Has  superior  knowledge  and/or  skill.  Operates  well 

independently  to  complete  all  assigned  tasks  in  a  timely 
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manner.  Recognizes  common  stimuli  variable  patterns  and 
applies  domain  rules  of  behavioral  responses.  Recognizes 
existence  of  new  variables  and/or  situations  in  which  the 
domain  rules  no  longer  apply,  and  can  develop  new  rules. 
Capable  of  solving  difficult  or  complex  problems. 

The  goal  of  the  training  in  Block  I  AWACS  WD  training  is  to  bring  all  students  up 
to  the  NOVICE/level  2for  Procedural  Knowledge,  Declarative  Knowledge,  and  Operational 
skill. 


COURSE  STRUCTURE 

Since  the  Knowledge  Domain  consists  largely  of  Procedural  Knowledge,  the 
required  step  sequences  are  already  structured.  Yet  there  is  a  need  for  a  comprehensive 
course  outline  that  addresses  the  goals  of  Procedural  Knowledge  to  be  taught  in  each 
lesson,  how  many  lessons  there  will  be,  and  how  fast  the  pace  of  presentation  will  be. 
The  course  outline  is  currently  taken  from  the  list  of  switch  actions  provided  by  the 
552  AWACW/DOQMW  office  and  the  IQT-WD  Reading  Guide.  The  maximum  time 
allowed  for  the  course  is  specified,  but  no  minimum  time  is  established. 

Student  Skills  Assessment 

For  this  proof  of  concept  prototype,  we  developed  a  Student  Skills  Assessment 
module  to  categorize  student  WDs  (from  1 -Novice  to  5-Master)  according  to  experience 
and  ability.  This  module  consists  of  an  Experience  Questionnaire  and  a  Switch  Action 
Exercise  that  students  complete  on  the  console  prior  to  the  first  lesson.  A  printed  copy 
of  this  questionnaire  is  included  as  Appendix  B.  Although  it  is  highly  probable  that  the 
students  taking  this  course  are  novice  WDs  just  graduated  from  the  introductory  course 
at  Tyndall  AFB,  exceptions  occur  with  sufficient  frequency  that  they  must  be  addressed. 
For  example,  occasionally  a  student  may  have  extensive  experience  either  as  a  WD  or  as 
an  aircraft  controller  and  must  be  requalified  as  required  by  regulations.  The  multiple- 
choice  questionnaire  ascertains  the  same  information  usually  gleaned  from  the  student’s 
records  or  informally  gathered  by  the  instructors  during  the  lesson  breaks.  In  addition, 
where  the  student  indicates  a  proficiency  above  basic  entry  level,  the  Switch  Action 
Zeroise  is  presented  to  validate  the  evaluation.  The  normal  Lesson  One,  Block  I 
instruction  is  then  presented. 

The  first  lesson  consists  of  three  parts  and  is  presented  at  only  one  pace  of 
instruction.  The  Instructor  Model  does  not  evaluate  the  student  during  the  lesson,  only 
after  the  lesson  is  complete.  The  capability  to  evaluate  the  student  during  the  lesson  and 
the  capability  to  change  the  lesson  mode,  If  necessary,  need  to  be  more  fully  developed. 
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Lesson  One 


The  Simulator  Model  is  the  means  of  instructional  presentation  and  includes  five 
phases  of  instruction  in  this  high  performance  Knowledge  Domain  (Fink,  1990): 

(1)  Static  overview  knowledge, 

(2)  General  procedure-oriented  knowledge, 

(3)  Guided  example  exercises, 

(4)  Unguided  example  exercises,  and 

(5)  Automated  example  exercises. 

Static  overview  knowledge  consists  of  a  general  description  of  the  salient  parts  and 
features  of  the  particular  piece  of  equipment  on  which  the  task  will  be  performed.  This 
static  overview  is  not  included  in  the  prototype.  Instead,  it  is  left  to  the  written  material 
found  in  the  IQT  WD  Student  Study,  Vol.  I. 

General  procedure-oriented  knowledge  consists  of  a  description  of  the  steps  that 
must  be  performed  in  executing  the  procedure  being  taught.  Parts  of  the  equipment 
involved  and  the  motivation  or  effect  for  each  step  are  indicated.  Each  of  the  lessons 
should  start  here  for  the  lowest  level  student.  The  first  part  of  Lesson  One  describes  the 
steps  and  indicates  where  the  switches  are.  it  doesn’t  give  any  motivations  for  each  step 
or  each  step’s  effect  on  the  overall  status  of  the  goal. 

Guided  example  exercises  provide  the  student  with  the  opportunity  to  practice  with 
specific  examples  while  being  prompted  and  coached  in  order  to  develop  accuracy  in  the 
skill.  The  second  part  of  Lesson  One  presents  a  procedure  the  student  must  perform  on 
the  Simulator.  Instructions  guiding  the  student  are  presented,  but  the  student  is  not 
evaluated  during  the  lesson. 

Unguided  example  exercises  provide  the  student  the  opportunity  to  practice  the 
whole  process  with  specific  examples  without  interruption  in  order  to  develop  speed  with 
accuracy.  These  exercises  comprise  the  third  and  final  part  of  the  Lesson  One.  The 
student  is  given  10  min  to  execute  the  Console  Checkout  while  reading  and  following  the 
Console  Checkout  checklist.  After  10  min,  the  lesson  is  terminated  and  the  data  on 
student  performance  is  gathered  and  evaluated,  in  future  ITS  development,  this  process 
should  be  automated. 

Automated-example  exercises  provide  the  student  the  opportunity  to  practice  the 
entire  process  with  spe^c  examples  while  doing  another  task.  These  types  of  exercises 
allow  the  WD  to  develop  the  capability  to  perform  tasks  automatically.  The  first  lesson 
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is ‘not  designed  to  use  this  phase.  However,  Console  Checkout  is  done  at  the  beginning 
of  each  and  every  lesson.  As  the  lessons  proceed,  the  time  for  this  procedure’s 
completion  will  narrow  down  from  10  min  to  3  min. 


DEVELOPING  THE  SOFTWARE 

The  development  of  this  prototype  was  accomplished  with  a  minimum  amount  of 
coding  change  to  the  existing  AWACS  Simulation  software.  Rather  than  having  the  two 
pieces  of  software  integrated,  the  ITS  software  executes  as  a  separate  process  from  the 
simulation,  but  uses  the  data  collected  in  the  Logger  File  as  input  for  evaluation.  The 
intent  is  to  have  both  pieces  of  software  execute  in  an  alternating  fashion  so  the  results 
of  one  can  be  used  in  the  next  execution  of  the  other.  Hence,  the  ITS  software  brackets 
the  simulation  to  provide  the  appearance  of  an  imbedded  simulation  without  actually 
accomplishing  the  imbedding  process.  On  the  front  end,  the  ITS  software  provides  the 
interaction  necessary  to  instruct  and  evaluate  the  student  before  selecting  an  appropriate 
scenario  for  the  AWACS  Simulation.  Then  on  the  back  end,  the  ITS  software  evaluates 
the  student’s  performance  during  the  simulation  before  presenting  the  appropriate  follow- 
on  instruction  area. 

Admittedly,  using  two  computers  systems  to  provide  the  functionality  for  the 
AWACS  ITS  is  not  the  ideal  solution.  However,  it  does  provide  a  flexible  test  platform 
upon  which  to  develop  some  quantitative  metrics  for  estimating  the  skills  of  the  student 
WD.  While  pedagogy  suggests  the  development  of  high-performance  skill  is  best 
accomplished  via  consistent  practice,  establishing  the  criteria  to  judge  the  WD  student  still 
requires  significant  amounts  of  definition  and  refinement. 


AWACS  ITS  PROGRAM  DOCUMENTATION 

The  software  for  the  AWACS  ITS  was  developed  on  both  the  UNIX  based  Silicon 
Graphics  (SG)  4D/50  and  the  DEC  VAX  780  computers  using  C  and  the  Curses  screen 
handling  package.  With  the  exception  of  the  simulation  graphics  (SG  only),  all  user 
interaction  can  be  accomplished  using  either  the  SG  graphics  or  DEC  VT200  series 
terminals. 

The  AWACS  ITS  operates  as  three  separate  parts: 

1.  The  SG  4D/50  software  provides  the  instructional  capability  for  the 
ITS. 

2.  The  Simulation  software  provides  the  hands-on  aspect  of  the  WD’s 
console  operation. 
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3.  The  VAX  780  software  is  the  first  iterative  trial  to  evaluate  a  WD’s 
actual  console  operation. 

SRL  provided  program  documentation  for  the  AWACS  ITS  software  in  contract 
deliverables  to  USAF  Armstrong  Laboratory,  Crew  Technology  Division,  Sustained 
Operations  Branch  (AL/CFTO).^  An  ITS  User’s  Guide  is  included  as  Appendix  C  of  this 
paper. 


RECOMMENDED  DEVELOPMENT 

After  successfully  completing  the  proof-of-concept,  several  areas  could  be  further 
developed  depending  on  future  funding  levels.  The  following  are  suggested  refinements 
to  the  existing  software: 

1 .  Besides  determining  the  start  and  stop  of  each  switch  action,  each  of  the 
associated  switch-checking  modules  could  be  made  considerably  more 
intelligent  (e.g.,  devise  methods  of  linking  such  things  as  time  delays  and 
erroneous  inputs  with  help  messages  and  switch  menus).  This 
development  would,  in  effect,  provide  an  interim  capability  for  both  the 
student  and  instructor  to  review  performance. 

2.  Provide  a  better  prompt  for  Control  key  paging  functions.  This  procedure 
may  entail  just  displaying  paging  options  in  appropriate  places  on  the 
screen,  i.e.,  Ctrl-P,  up-arrow,  Ctrl-B  at  the  top  of  the  screen  and  Ctrl-N, 
down-arrow,  Ctrl-F  at  the  bottom  of  the  screen. 

3.  Options  to  develop: 

a.  Establish  criteria  for  those  values  that  are  to  be  scored  and  stored. 

b.  Provide  a  mechanism  for  the  student  to  review  material  and  answers, 
but  preclude  the  capability  to  change  original  values. 

c.  Establish  criteria  for  demotion,  promotion,  or  instructor  intervention 
based  upon  scores. 

4.  Build  a  utility  to  assist  in  the  development  of  a  lesson  and/or  instruction 
block  so  the  instructor’s  attention  can  be  focused  on  content  and  not 
procedure. 


‘  Systems  Research  Laboratories,  Inc.:  "Research  and  Development  Computer  Software  Report,  Delivery 
Order  0008,  Attachment  2,  Sequence  1,"  Contract  No.  F33615-87-D-0601,  September  1990. 
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While  these  changes  would  significantly  improve  the  system’s  usability,  additional 
development  is  required  to  produce  a  fully  functional  AWACS  ITS.  These  improvements 
should  include: 

1 .  Integrate  the  AWACS  Simulation  and  ITS  software  into  a  single  package  so 
that  it  will  qualify  as  an  expert  system  that  is  limited  to  switch  action 
procedures.  This  limitation  obviates  the  knotted  problems  that  are  often 
associated  with  the  inclusion  of  strategy. 

For  the  start  of  this  effort,  the  current  ITS  operation  will  be  continued,  i.e., 
maintain  execution  in  batch  fashion,  and  improve  the  software  so  it  can 
reliably  detect  the  beginning  and  ending  responses  to  events.  In  turn,  each 
established  detection  scheme  will  be  coded  into  the  Simulation  software  so 
that  the  end  result  should  be  the  limited  expert  system  that  is  desired. 

2.  Using  the  just  described,  limited  expert  system  as  a  kernel,  this 
development  would  add  the  capability  to  detect  and  suggest  solutions  to 
switch  action  procedural  problems  while  the  Simulation  is  running. 

With  the  knowledge  that  the  Simulation  software  controls  the  scenario 
presentation  to  the  student  and  that  it  can  now  determine  the  correctness 
of  switch  action  procedures,  it  is  now  possible  to  determine  the 
appropriateness  of  proffered  switch  action  procedures  in  response  to 
corrtrolled  scenario  situations.  This  development  would  allow  the  software 
to  passively  monitor  a  student’s  actions  and  provide  real-time  diagnostics 
as  procedural  errors  occur. 

3.  Incorporate  a  reactive  Instructor  and  Student  Model  into  the  software  that 
was  developed  during  Step  2. 

Since  the  software  can  now  monitor  a  student’s  switch  actions  to  known 
situations,  this  would  begin  the  development  of  both  the  Instructor  and 
Student  Model  software.  Essentially,  the  software  will  be  given  the  capability 
to  evaluate  a  student  and  determine  whether  the  student  requires  either 
advancement,  remediation,  or  to  be  kept  at  status  quo.  These  software 
decisions  will  be  based  upon  actions  that  were  taken.  Consideration  of 
actions  that  were  omitted  will  be  avoided  unless  outcomes  are  blatantly 
obvious. 
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Appendix  A 
IQT  Block  I 

Topics  and  Switch  Actions 
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Day  1 

Familiarization  and  Mechanics 
Assign  Console 

TD  Index 

Hardcopy 

TD  Update 

Day  2 

Line 

Circle 

Coordinates 

Arrow 

Message 

Day  3 

Winds  Aloft  TD 

Airbase  Weather  TD 

Initialize  Special  Point 
RN/DES/NTN  SD 

Day  4 

Identifying  Tracks 

Initiate 

Reinitiate 

Mode  IV 

Drop 

Assign/Defer 

Request  SIF 

Locate  SIF 

Day  5 

Track  Blocks 

Track  TDs/Special  Point  TDs 

Day  6 

Radar/IFF  Tracking 
Request/Assign  IFF/SIF 

Area  Define/Delete 

Corridor  IFF  SD 

Day  7 

RCT  Initialization  TD 
Add/Delete  Airbase 

Commit 

Cap 

RTB 

Alter  Control 
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Day  8 

RCT  Mission/Augmented  Msn  TD 
Manual  Guidance 
Fuel/Configuration 

Armament  Update/Override 
Command  Tracking 

RCT  Mode  Control 

Aircraft  Down 

Day  9 

Mission  Modifier 

Recovery  Airbase 

Beam  Rider 

Bearing  and  Range 

Tactical  Bearing  and  Range 

Day  10 

Order  of  Battle  Add/Delete 

Order  of  Battle  SD 

Restricted  Area 

Present  Altitude 

Target  Altitude 

Day  11 

Intercept  Line  SD 

MA/WII 

UHF  Tune 

WT  Handover 

Accept/Reject  Handover 
Wilco/Cantco 

Day  12 

Review  All  Block  1  Academics 

Day  13 

Written  Evaluation  and  Critiques 
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Appendix  B 

Experience  Questionnaire 
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EXPERIENCE  QUESTIONNAIRE 


1.  Have  you  ever  been  Mission  Ready  (MR)  qualified  as  a  17XX  ? 

A.  Yes 

B.  No 

2.  Were  you  MR  as  a _ ? 

A.  U.S.  or  NATO  E-3  WD/IWD/SD/ISD 

B.  U.S.  or  NATO  E-3  ASO 

C.  Other 

3.  You  are  here  because _ . 

A.  You’ve  been  DNIF  for  more  than  180  days. 

B.  You  failed  an  EVAL  and  are  being  retrained. 

C.  You’ve  only  been  NATO  qualified. 

4.  Do  you  have  any  other  17XX  experience  ? 

A.  Yes 

B.  No 

5.  Your  other  17XX  experience  includes _ . 

A.  MCE 

B.  407L  CRC/CRP  WC/WAO 

C.  407L  CRC/CRP  ASO 

D.  NORAD/ICELAND/ALASKA  ROCC/SOCC  WC/WAO 

E.  NORAD/ICELAND/ALASKA  BOCC/SOCC  ASO 

F.  Manual  Radar  (FACP/TSP-43E) 

G.  Other 

6.  Does  your  other  17XX  experience  include  a  stint  as  an  exchange  officer  with  the 
U.S.  Navy  in  the  E-2C  ? 

A.  Yes 

B.  No 
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7.  Did  you  graduate  from  Tyndall’s  Basic  WC  Course  (E30BP-1741A-004)? 

A.  Yes 

B.  No 

8.  Were  you _ ? 

A.  a  Distinguished  Graduate 

B.  in  the  top  25%  of  your  class 

C.  average 

D.  barely  passed 

9.  Did  you  take  the  automated  course? 

A.  Yes 

B.  No 

10.  Vy/ere  you  prior  enlisted? 

A.  U.S.  Air  Force 

B.  U.S.  Navy 

C.  U.S.  Army 

D.  U.S.  Marine 

E.  No 

1 1 .  Are  you  a  former  1 6XX  ? 

A.  Yes 

B.  No 

12.  While  you  were  prior  enlisted  in  the  Air  Force,  did  you  have  the  AFSC _ 

A.  11XXX 

B.  276XX 

C.  30XXX  (Air  Traffic  Controller  at  a  RAPCON) 

D.  Other 

13.  While  you  were  prior  enlisted  in  the  Army,  did  you  have  a  previous  job  as 

A.  Helicopter  pilot 

B.  Fire  control  officer  for  Patriot  or  l-HAWK  battery 

C.  Radar  Air  Traffic  Controller 
0.  Joint  Stars  operator 

E.  Other 
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14.  While  you  were  prior  enlisted  in  the  Navy,  did  you  have  a  previous  job  in 


A.  Radar  Air  Traffic  Control 

B.  Air  Intercept  Control 

C.  Other 

15.  While  you  were  prior  enlisted  in  the  Marines,  did  you  have  a  previous  job  in 
? 


A.  Radar  Air  Traffic  Control 

B.  Air  Intercept  Control 

C.  Other 

16.  Are  you  an  exchange/liaison  officer  from _ ? 

A.  the  U.S.  Army 

B.  the  U.S.  Navy 

C.  the  U.S.  Marines 

D.  another  nation 

17.  In  the  U.S.  Army,  your  previous  jobs  included _ 

A.  Pilot 

B.  Joint  Stars 

C.  Air  Defense  Artillery 

D.  Radar  Air  Traffic  Control 

E.  ASOC  (Air  Support  Operations  Center) 

F.  Other 

18.  In  the  U.S.  Navy,  your  previous  jobs  Included _ . 

A.  Pilot 

B.  Radar  Intercept  Officer 

C.  E-2C  Naval  Right  Officer 

D.  Radar  Air  Traffic  Control 

E.  Air  Intercept  Control 

F.  Other 
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19.  In  the  U.S.  Marines,  your  previous  jobs  included _ 

A.  Pilot 

B.  Air  Defense  Artillery 

C.  Radar  Air  Traffic  Control 

D.  F-4  Weapons  System  Operator  (WSO) 

E.  MACS  5/6/7  Radar  Control/ Air  Intercept  Control 

F.  Other 

20.  You  are  a _ officer. 

A.  United  Kingdom 

B.  Canadian 

C.  French 

D.  German 

E.  Other 

21.  You  have  experience  in/as _ . 

A.  Pilot/Navigator 

B.  Weapons  System  Officer  (WSO)  in  F-4/Tornado  IDS 

C.  Schackelton  AEW.2 

D.  NATO  E-3  or  UK  E-3D  AEW.Mk.1  ASO 

E.  NATO  E-3  or  UK  E-3D  AEW.Mk.1  WC/WAO 

F.  Other  Radar  Control  Air  Defense  Work 

G.  Other  (work  not  involving  Radar) 

22.  You  have  experience  in/as _ . 

A.  U.S.  E-3  ASO 

B.  NATO  E-3  WC/WAO 

C.  NATO  E-3  ASO 

D.  NORAD  ROCC/SOCC 

E.  Ground  Radar  Weapons  Controller 

F.  Other 

23.  You  have  experience  in/as _ . 

A.  Pilot/Navigator 

B.  NATO  E-3  WC/WAO 

C.  NATO  E-3  ASO 

D.  NADGE  WC 

E.  Other 
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24.  You  have  experience  in/as _ . 

A.  Pilot/Navigator 

B.  French  E-3F  WC/WAO 

C.  French  E-3F  ASO 

D.  Other 

25.  Are  you  a  Fighter  Weapons  Instructor  Course  graduate? 

A.  Yes 

B.  No 
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Appendix  C 
ITS  User’s  Guide 
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Intelligent  Tutoring  System 
User’s  Guide 


You  may  run  the  ITS  program  using  a  SGI  terminal  even  though  its  software  is  spread 
across  a  SGI/UNIX  and  DEC/VMS  based  system.  Functionally,  the  software  is  used  as 
follows: 


•  The  presentation  of  the  questionnaire  function  is  accomplished  using  only 
the  SGI/UNIX  system, 

•  The  presentation  of  the  Simulation  is  accomplished  using  a  combination  of 
both  the  SGI/UNIX  and  DEC/VMS  systems. 

•  The  presentation  of  the  evaluation  software  is  accomplished  using  only  the 
DEC/VMS  system. 

Each  function  is  started  separately.  While  having  to  start  each  function  is  not  ideal,  the 
modularity  did  provide  a  better  software  development  environment  with  no  impact  to  the 
existing  Simulation  software. 


Questionnaire  Function 

To  begin  the  Questionnaire  Function,  you  must  have  the  "ITS  Window"  on  the  screen. 
This  window  is  sized  to  an  80  character  by  24  line  (80x24)  display  for  the  presentation  of 
questionnaire  material.  Enter  the  following  command  to  initiate  this  window: 

wsh  -n  MTS  Window'  -p1 75,300  -s80,24 

Enter  ITS  at  the  prompt  within  the  window. 

ITS  prompts  you  as  follows: 

1. 

Use  cursor  to  select: 

Instructor  Student 
then  press  'Enter' 

The  desired  selection  is  highlighted  on  the  screen, 
if  you  select  the  Instructor  option,  continue  with  Step  2. 

If  you  select  the  Student  option,  continue  with  Step  13. 
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2. 


At  this  point,  nothing  on  the  screen  changes.  However,  ITS  expects  you  to 
enter  the  proper  password  before  continuing.  Entry  of  any  other  value 
causes  the  program  to  terminate  without  any  indication.  Once  you  enter  the 
proper  password,  continue  with  Step  3. 


3.  ITS  displays  the  following  prompt: 

Enter  index  of  desired  option: 

1 .  Add  an  instruction  block 

2.  Delete  an  instruction  block 

3.  Modify  an  instruction  block 

4.  Review  an  instruction  block 

5.  Review  a  student’s  performance 

6.  Adjust  a  student’s  instruction 

Selection: 

Enter  the  desired  option  by  specifying  1,  2  etc. 

Note:  Only  options  1-3  are  implemented.  Selections  4-6  are  not 

implemented  at  this  time. 

Enter  one  of  the  options  1-3,  continue  with  Step  4. 

Note:  If  you  enter  any  value  other  than  1-6,  ITS  displays  Invalid  response. 
Press  the  Enter  key  to  acknowledge  and  ITS  displays  the  following 
message: 

Terminate  ITS  [Y/N]: 

If  you  enter  any  value  other  than  Y,  ITS  interprets  it  as  a  N  response  and 
returns  to  the  start  of  this  Step.  If  you  enter  Y,  the  program  terminates. 


4.  ITS  prompts  you  for  the  specification  of  the  path  to  the  Instructor  directory. 

However,  if  you  have  already  accomplished  this  Step  (on  a  previous  pass 
through  the  program),  ITS  continues  with  Step  5.  If  not,  the  following 
display  appears: 

Enter  path  to  "Instructor"  file  directory 
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You  may  either  specify  the  Instructor  file  directory  or  press  the  Enter  key 
to  get  the  default  value  as  stipulated  by  instr_path_def  in  itsdef.h.  In  both 
cases  ITS  prompts  you  to  verify  the  specification: 

Is  path  correct?  [Y/N/Q]: 

If  you  enter  Q,  the  program  terminates. 

If  you  enter  N,  this  Step  is  repeated. 

If  you  enter  Y,  continue  with  Step  5. 


5.  ITS  displays  titles  of  existing  instruction  Blocks  along  with  the  following 
message: 

Use  cursor  to  position,  'Enter'  to  select,  or  'Q'  to  quit 

If  you  selected  option  1  in  Step  3,  Add  an  Instruction  Block,  ITS  displays 
a  reminder  of  existing  titles.  For  options  2  or  3,  you  may  choose  the 
Instruction  Block  that  is  to  be  modified  or  deleted,  respectively. 

If  you  selected  option  1  in  Step  3,  continue  with  Step  6. 

If  you  selected  option  2,  Delete  an  instruction  block,  ITS  continues  with 
Step  9. 

If  you  selected  option  3,  Modify  an  instruction  block,  ITS  continues  with 
Step  12. 


6.  ITS  asks  you  to  enter  a  title  to  the  new  instruction  block: 

Enter  title  (max  =  80  chars): 

Enter  the  title  and  continue  with  Step  7. 
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7.  ITS  asks  you  if  there  are  any  lessons  associated  with  this  Instruction  Block; 

Any  lessons  in  this  instruction  block  [Y/N]: 

You  may  create  the  instruction  block  entry  by  entering  N  and  ITS  continues 
with  Step  3,  or  if  lessons  are  to  be  added,  enter  Y  and  continue  with  Step 
8. 


8.  ITS  displays  the  title  of  the  instruction  block  and  a  numbered  list  of  empty 
lesson  entries,  followed  by  the  prompt: 

Enter  lesson  name  (max  =  10  char): 

Enter  the  names  of  the  lessons  that  are  to  comprise  this  instruction  block. 
When  you  have  finished,  terminate  this  Step  by  entering  q  to  lesson  name 
and  ITS  continues  with  Step  3. 


9.  ITS  displays  the  title  of  the  instruction  block  and  a  numbered  list  of  the 
lessons  in  the  block  along  with  the  following  prompt: 

Do  you  want  to  delete  lessons?  [Y/N]: 

If  you  want  to  delete  only  the  Instruction  Block,  enter  N  and  ITS  continues 
with  Step  3. 

If  you  also  want  to  delete  lesson  files,  enter  Y  and  continue  with  Step  10. 


10.  ITS  displays  the  following  prompt: 

Delete  (a}ll  or  s(ome)  of  these  lessons: 

If  you  want  to  delete  ail  the  lesson  files  listed,  enter  a  and  ITS  continues 
vrith  Step  3. 

If  you  want  to  selectively  delete  some  of  the  lesson  files,  enter  s  and 
continue  with  Step  11. 
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11.  ITS  displays  the  following  prompt: 

Enter  index  of  lesson  to  be  deleted: 

Enter  the  number  that  appears  with  the  lesson  name. 

Note:  The  numbered  list  is  reordered  after  each  deletion  so  indices  can 
change  for  each  specified  deletion. 

When  you  are  finished  deleting,  enter  q  and  ITS  continues  with  Step  3. 


12.  ITS  displays  the  title  of  the  instruction  block  and  a  numbered  list  of  the 
lessons  in  the  block  along  with  the  following  prompt: 

Modifying  (t)itle  or  (l)esson: 

If  you  want  to  modify  the  title  of  the  instruction  block,  enter  t  and  the 
prompt  Enter  title  will  overwrite  the  previous  prompt  and  allow  you  to 
specify  a  new  title.  ITS  continues  with  a  repetition  of  this  Step. 

If  you  want  to  modify  a  lesson,  enter  I  and  ITS  displays  the  following 
prompt: 

Enter  index  of  lesson  to  be  changed; 

Enter  the  number  associated  with  the  lesson  to  be  changed  and  ITS 
displays  the  following  prompt: 

Change:  (d)elete,  (m)odify,  (i)nsert: 

If  you  want  to  delete  the  selected  lesson,  enter  d.  ITS  continues  with  the 
repetition  of  this  Step. 

if  you  want  to  insert  or  modify  a  lesson,  enter  m  or  I.  ITS  displays  the 
prompt  Enter  lesson  name.  Modify  or  Insert  the  lesson  name  and  ITS 
continues  with  a  repetition  of  this  Step. 

To  terminate  this  Step,  enter  q  and  ITS  continues  with  Step  3. 
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1 3.  ITS  asks  you  to  specify  the  path  for  the  Instructor  and  Student  directories. 
If  one  or  both  use  the  default  values,  the  values  for  the  Instructor  and 
student  paths  are  taken  from  the  instr_path_def  or  the  stdt_path_def 
values  in  itsdef.h,  respectively. 


For  the  instructor  path,  ITS  displays: 

Enter  path  to  "Instructor"  file  directory 
-> 

For  the  student  path,  ITS  displays: 

Enter  path  to  "Student"  file  directory 
-> 

Both  are  followed  by  the  prompt: 

Is  path  correct?  [Y/N/Q]: 

If  you  enter  Q,  the  program  stops. 

If  you  enter  N,  this  Step  is  repeated. 

If  you  enter  Y,  then: 

a.  If  this  response  is  to  the  instructor  path  prompt,  ITS 
continues  with  a  request  for  the  student  path. 

b.  If  this  request  is  to  the  student  path  prompt,  ITS  continues 
with  Step  14. 
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14.  ITS  prompts  you  (the  student)  for  identification: 

Enter  name  and  SSAN: 

Name: 

ITS  asks  you  to  verify  identification: 

Is  name  spelled  correctly?  [Y/N] 

If  you  answer  N,  this  request  is  repeated. 

If  you  answer  Y,  ITS  requests  your  Social  Security  Number  (SSAN): 

SSAN:  xxx-xx-xxxx 
Verify  your  entry  by  answering: 

Is  SSAN  correct?  [Y/N] 

If  you  enter  N,  SSAN  portion  of  the  this  Step  is  repeated. 

If  you  enter  Y,  continue  with  Step  15. 


15.  If  the  student  is  new,  ITS  displays  the  following  verification  step: 

Are  you  a  new  student?  [Y/N] 

If  you  enter  N,  ITS  repeats  Step  14.  Otherwise,  if  the  student  is  either  new, 
as  Indicated  by  a  Y  reply,  or  has  not  completed  the  Experience 
Questionnaire,  as  indicated  in  the  student’s  database,  ITS  presents  the 
questionnaire. 

At  the  conclusion  of  the  questionnaire,  ITS  begins  the  Simulation.  If  the 
student  is  above  the  base  entry  level,  then  the  Simulation  is  used  to  present 
a  scenario  to  try  and  validate  the  determined  level.  Otherwise,  the 
Simulation  consists  of  the  Console  Checkout  lesson. 
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Simulation  Function 


Documentation  for  the  Simulation  Function  can  be  found  in: 

Systems  Research  Laboratories,  Inc.:  "Research  and  Development 
Computer  Software  Report,  Delivery  Order  0008,  Attachment  2, 
Sequence  1,"  Contract  No.  F33615-87-D*0601,  September  1990. 


Evaluation  Function 

To  execute  the  Evaluation  Function,  you  must  be  in  a  window  with  access  to  the  VAX. 
It  is  also  assumed  that  the  student  has  completed  a  session  with  the  Simulation. 

Prior  to  running  the  software  to  evaluate  the  student’s  performance,  the  data  captured  by 
the  Simulation  must  be  preprocessed  so  it  is  time  ordered.  This  is  accomplished  by 
executing  the  following  command  on  the  logger  file. 

reduce  "mindisk  0  f  1  I  6  sim  90  dir  [logger  file  directory  string]  status" 

go 

quit 

Upon  completion  of  the  REDUCE  run,  begin  the  evaluation  by  entering: 

run  sdt  evai 


Processing  continues  with  the  following  steps. 

1.  ITS  asks  you  to  specify  the  location  of  the  output  of  the  previously 
mentioned  REDUCE  process. 

Enter  name  of  the  data  file 
Name: 

Enter  the  catalog/file  string  of  the  Pass  6  logger  file  from  REDUCE.  Upon 
completion,  ITS  asks  you  to  verify  that  the  file  specified  is  a  REDUCE  pass 
6  output. 

2.  ITS  asks  you  to  specify  the  number  of  WDs  tested: 

Enter  the  number  of  WDs  that  were  tested: 
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3.  ITS  asks  you  to  associate  the  WDs  with  a  specific  console  number  by 
answering  the  following  query  for  the  each  WD  tested: 

Enter  console  no.  of  xxx  WD: 

4.  ITS  asks  you  to  specify  the  WD  id  number  for  each  console  that  contained 
a  tested  WD: 

Enter  WD  id  no.  for  console  x: 
where  x  is  substituted  with  the  appropriate  console  number. 

5.  For  this  Step,  ITS  displays  the  following: 

Select  skill  level  for  evaluation: 

1)  Naive 

2)  Novice 

3)  Journeyman  - 

4)  Expert 

5)  Master  ■" 

Selection: 

Enter  an  appropriate  value.  At  this  time,  this  value  is  not  used.  The  intent 
is  to  provide  multiple  criteria  when  evaluating  the  scenario  from  a  Simulation 
run. 

6.  ITS  asks  for  the  location  of  the  event  script  file: 

Enter  name  of  event  script  file 
Name: 

Upon  answering  the  name  of  the  event  script  file,  processing  continues  until 
the  event  script  is  exhausted.  As  the  absence  or  presence  of  each  event 
is  detected,  a  brief  message  about  the  condition  is  displayed.  You  must 
acknowledge  each  message  with  the  Enter  key  before  the  program 
continues.  Currently,  display  of  the  captured  data  is  limited  to  the  screen. 
However,  recording  these  data  to  a  file  could  be  easily  accomplished. 
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